{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1f1121d-7f49-4a2a-b0de-ceebfdadb11b",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Default Rate Estimation using LightGBM\n",
    "\n",
    "### Introduction\n",
    "As we known, `LightGBM` is a very popular machine learning library in the data competitions and industries because of its excellent effect and interpretability. In this notebook, we will use this library to build our binary classification model trained on dataset of [Tianchi Competetion](https://tianchi.aliyun.com/competition/entrance/531830/information), referencing some excellent work as below:\n",
    " * Feature Eningeering: https://tianchi.aliyun.com/notebook-ai/detail?spm=5176.12586969.1002.6.3b30b135z4zdwX&postId=129321\n",
    " * Hypermeter Tunning: https://tianchi.aliyun.com/notebook-ai/detail?spm=5176.12586969.1002.3.3b30b1352BkwCe&postId=129346\n",
    " \n",
    "### Requirements\n",
    "Suppose we have run the `../../../tianchi_loan/fg.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a4055f60-de65-4b73-9fb9-28d79ce45dde",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyspark\n",
    "import yaml\n",
    "import argparse\n",
    "import subprocess\n",
    "import lightgbm as lgb\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import warnings\n",
    "\n",
    "from lightgbm import Booster, LGBMClassifier\n",
    "from sklearn.model_selection import StratifiedKFold, KFold\n",
    "from sklearn.metrics import accuracy_score, f1_score, roc_auc_score, log_loss\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "pd.set_option('display.max_rows', None)\n",
    "pd.set_option('display.max_columns', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c0dc987-8a6e-49d7-af94-1a53c9348f4a",
   "metadata": {},
   "source": [
    "### Read Dataset\n",
    "\n",
    "In this section, we need read training dataset stored in `${MY_S3_BUCKET}/risk/tianchi/fg_train_data.csv`. You may need substitute `${MY_S3_BUCKET}` with your own S3 bucket before runing the **commented code** below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cb7f924d-dbe7-40fb-8313-c06e9d0795e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !aws s3 cp ${MY_S3_BUCKET}/risk/tianchi/fg_train_data.csv ../../../dataset/tianchi_loan/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b0b3eac1-2828-41b1-94fc-1daa076daa34",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = pd.read_csv('../../../dataset/tianchi_loan/fg_train_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b12e26bd-ae1f-4012-8411-1493e4793f87",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>loanAmnt</th>\n",
       "      <th>term</th>\n",
       "      <th>interestRate</th>\n",
       "      <th>installment</th>\n",
       "      <th>grade</th>\n",
       "      <th>subGrade</th>\n",
       "      <th>employmentTitle</th>\n",
       "      <th>employmentLength</th>\n",
       "      <th>homeOwnership</th>\n",
       "      <th>annualIncome</th>\n",
       "      <th>verificationStatus</th>\n",
       "      <th>isDefault</th>\n",
       "      <th>purpose</th>\n",
       "      <th>postCode</th>\n",
       "      <th>regionCode</th>\n",
       "      <th>dti</th>\n",
       "      <th>delinquency_2years</th>\n",
       "      <th>ficoRangeLow</th>\n",
       "      <th>ficoRangeHigh</th>\n",
       "      <th>openAcc</th>\n",
       "      <th>pubRec</th>\n",
       "      <th>pubRecBankruptcies</th>\n",
       "      <th>revolBal</th>\n",
       "      <th>revolUtil</th>\n",
       "      <th>totalAcc</th>\n",
       "      <th>initialListStatus</th>\n",
       "      <th>applicationType</th>\n",
       "      <th>earliesCreditLine</th>\n",
       "      <th>title</th>\n",
       "      <th>policyCode</th>\n",
       "      <th>n0</th>\n",
       "      <th>n1</th>\n",
       "      <th>n2</th>\n",
       "      <th>n3</th>\n",
       "      <th>n4</th>\n",
       "      <th>n5</th>\n",
       "      <th>n6</th>\n",
       "      <th>n7</th>\n",
       "      <th>n8</th>\n",
       "      <th>n9</th>\n",
       "      <th>n10</th>\n",
       "      <th>n11</th>\n",
       "      <th>n12</th>\n",
       "      <th>n13</th>\n",
       "      <th>n14</th>\n",
       "      <th>issueDateDT</th>\n",
       "      <th>grade_target_mean</th>\n",
       "      <th>subGrade_target_mean</th>\n",
       "      <th>grade_to_mean_n0</th>\n",
       "      <th>grade_to_std_n0</th>\n",
       "      <th>grade_to_mean_n1</th>\n",
       "      <th>grade_to_std_n1</th>\n",
       "      <th>grade_to_mean_n2</th>\n",
       "      <th>grade_to_std_n2</th>\n",
       "      <th>grade_to_mean_n4</th>\n",
       "      <th>grade_to_std_n4</th>\n",
       "      <th>grade_to_mean_n5</th>\n",
       "      <th>grade_to_std_n5</th>\n",
       "      <th>grade_to_mean_n6</th>\n",
       "      <th>grade_to_std_n6</th>\n",
       "      <th>grade_to_mean_n7</th>\n",
       "      <th>grade_to_std_n7</th>\n",
       "      <th>grade_to_mean_n8</th>\n",
       "      <th>grade_to_std_n8</th>\n",
       "      <th>grade_to_mean_n9</th>\n",
       "      <th>grade_to_std_n9</th>\n",
       "      <th>grade_to_mean_n10</th>\n",
       "      <th>grade_to_std_n10</th>\n",
       "      <th>grade_to_mean_n11</th>\n",
       "      <th>grade_to_std_n11</th>\n",
       "      <th>grade_to_mean_n12</th>\n",
       "      <th>grade_to_std_n12</th>\n",
       "      <th>grade_to_mean_n13</th>\n",
       "      <th>grade_to_std_n13</th>\n",
       "      <th>grade_to_mean_n14</th>\n",
       "      <th>grade_to_std_n14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35000.0</td>\n",
       "      <td>5</td>\n",
       "      <td>19.52</td>\n",
       "      <td>917.97</td>\n",
       "      <td>5</td>\n",
       "      <td>21</td>\n",
       "      <td>161280</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>110000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>43</td>\n",
       "      <td>32</td>\n",
       "      <td>17.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>730.0</td>\n",
       "      <td>734.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24178.0</td>\n",
       "      <td>48.9</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2001</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2587</td>\n",
       "      <td>0.386234</td>\n",
       "      <td>0.380444</td>\n",
       "      <td>1.876011</td>\n",
       "      <td>3.992386</td>\n",
       "      <td>1.874620</td>\n",
       "      <td>4.053876</td>\n",
       "      <td>1.942294</td>\n",
       "      <td>4.023418</td>\n",
       "      <td>1.869160</td>\n",
       "      <td>3.948124</td>\n",
       "      <td>1.897562</td>\n",
       "      <td>4.055665</td>\n",
       "      <td>1.865760</td>\n",
       "      <td>4.017884</td>\n",
       "      <td>1.840872</td>\n",
       "      <td>4.074681</td>\n",
       "      <td>1.851544</td>\n",
       "      <td>4.040923</td>\n",
       "      <td>1.938318</td>\n",
       "      <td>4.024912</td>\n",
       "      <td>1.842210</td>\n",
       "      <td>4.108917</td>\n",
       "      <td>1.852810</td>\n",
       "      <td>4.009823</td>\n",
       "      <td>1.852810</td>\n",
       "      <td>4.009823</td>\n",
       "      <td>1.857394</td>\n",
       "      <td>4.005352</td>\n",
       "      <td>1.856379</td>\n",
       "      <td>3.991791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>18000.0</td>\n",
       "      <td>5</td>\n",
       "      <td>18.49</td>\n",
       "      <td>461.90</td>\n",
       "      <td>4</td>\n",
       "      <td>16</td>\n",
       "      <td>89538</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>46000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>64</td>\n",
       "      <td>18</td>\n",
       "      <td>27.83</td>\n",
       "      <td>0.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>704.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15096.0</td>\n",
       "      <td>38.9</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2002</td>\n",
       "      <td>5768</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1888</td>\n",
       "      <td>0.304227</td>\n",
       "      <td>0.298190</td>\n",
       "      <td>1.500809</td>\n",
       "      <td>3.193909</td>\n",
       "      <td>1.502905</td>\n",
       "      <td>3.185919</td>\n",
       "      <td>1.504054</td>\n",
       "      <td>3.173189</td>\n",
       "      <td>1.567352</td>\n",
       "      <td>3.204484</td>\n",
       "      <td>1.511316</td>\n",
       "      <td>3.139166</td>\n",
       "      <td>1.515599</td>\n",
       "      <td>3.098975</td>\n",
       "      <td>1.500817</td>\n",
       "      <td>3.139721</td>\n",
       "      <td>1.517874</td>\n",
       "      <td>3.086106</td>\n",
       "      <td>1.504140</td>\n",
       "      <td>3.174194</td>\n",
       "      <td>1.484104</td>\n",
       "      <td>3.173687</td>\n",
       "      <td>1.482248</td>\n",
       "      <td>3.207858</td>\n",
       "      <td>1.482248</td>\n",
       "      <td>3.207858</td>\n",
       "      <td>1.485915</td>\n",
       "      <td>3.204282</td>\n",
       "      <td>1.485103</td>\n",
       "      <td>3.193433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12000.0</td>\n",
       "      <td>5</td>\n",
       "      <td>16.99</td>\n",
       "      <td>298.17</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>159367</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>74000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>265</td>\n",
       "      <td>14</td>\n",
       "      <td>22.77</td>\n",
       "      <td>0.0</td>\n",
       "      <td>675.0</td>\n",
       "      <td>679.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4606.0</td>\n",
       "      <td>51.8</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2006</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3044</td>\n",
       "      <td>0.304227</td>\n",
       "      <td>0.302541</td>\n",
       "      <td>1.500809</td>\n",
       "      <td>3.193909</td>\n",
       "      <td>1.360761</td>\n",
       "      <td>2.998190</td>\n",
       "      <td>1.532981</td>\n",
       "      <td>3.241462</td>\n",
       "      <td>1.273891</td>\n",
       "      <td>3.071276</td>\n",
       "      <td>1.162371</td>\n",
       "      <td>3.176718</td>\n",
       "      <td>1.480241</td>\n",
       "      <td>3.125317</td>\n",
       "      <td>1.472698</td>\n",
       "      <td>3.259745</td>\n",
       "      <td>1.406712</td>\n",
       "      <td>3.254085</td>\n",
       "      <td>1.530998</td>\n",
       "      <td>3.244609</td>\n",
       "      <td>1.504230</td>\n",
       "      <td>3.089208</td>\n",
       "      <td>1.482248</td>\n",
       "      <td>3.207858</td>\n",
       "      <td>1.482248</td>\n",
       "      <td>3.207858</td>\n",
       "      <td>1.485915</td>\n",
       "      <td>3.204282</td>\n",
       "      <td>1.315111</td>\n",
       "      <td>3.146801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2050.0</td>\n",
       "      <td>3</td>\n",
       "      <td>7.69</td>\n",
       "      <td>63.95</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>59830</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>35000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>465</td>\n",
       "      <td>14</td>\n",
       "      <td>17.49</td>\n",
       "      <td>0.0</td>\n",
       "      <td>755.0</td>\n",
       "      <td>759.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3111.0</td>\n",
       "      <td>8.5</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2006</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2679</td>\n",
       "      <td>0.059838</td>\n",
       "      <td>0.065532</td>\n",
       "      <td>0.375202</td>\n",
       "      <td>0.798477</td>\n",
       "      <td>0.368239</td>\n",
       "      <td>0.796491</td>\n",
       "      <td>0.383245</td>\n",
       "      <td>0.810366</td>\n",
       "      <td>0.380622</td>\n",
       "      <td>0.806605</td>\n",
       "      <td>0.384972</td>\n",
       "      <td>0.802575</td>\n",
       "      <td>0.368526</td>\n",
       "      <td>0.819126</td>\n",
       "      <td>0.369865</td>\n",
       "      <td>0.798404</td>\n",
       "      <td>0.377964</td>\n",
       "      <td>0.799464</td>\n",
       "      <td>0.382750</td>\n",
       "      <td>0.811152</td>\n",
       "      <td>0.370128</td>\n",
       "      <td>0.799459</td>\n",
       "      <td>0.370562</td>\n",
       "      <td>0.801965</td>\n",
       "      <td>0.370562</td>\n",
       "      <td>0.801965</td>\n",
       "      <td>0.371479</td>\n",
       "      <td>0.801070</td>\n",
       "      <td>0.344287</td>\n",
       "      <td>0.793451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11500.0</td>\n",
       "      <td>3</td>\n",
       "      <td>14.98</td>\n",
       "      <td>398.54</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>85242</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>30000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>32.60</td>\n",
       "      <td>0.0</td>\n",
       "      <td>665.0</td>\n",
       "      <td>669.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>14021.0</td>\n",
       "      <td>59.7</td>\n",
       "      <td>33.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1994</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2406</td>\n",
       "      <td>0.224522</td>\n",
       "      <td>0.224686</td>\n",
       "      <td>1.125607</td>\n",
       "      <td>2.395431</td>\n",
       "      <td>1.113406</td>\n",
       "      <td>2.430896</td>\n",
       "      <td>1.133984</td>\n",
       "      <td>2.439745</td>\n",
       "      <td>1.121496</td>\n",
       "      <td>2.368874</td>\n",
       "      <td>1.197930</td>\n",
       "      <td>2.401168</td>\n",
       "      <td>1.120956</td>\n",
       "      <td>2.388727</td>\n",
       "      <td>1.106851</td>\n",
       "      <td>2.450979</td>\n",
       "      <td>1.144817</td>\n",
       "      <td>2.403154</td>\n",
       "      <td>1.133458</td>\n",
       "      <td>2.441340</td>\n",
       "      <td>1.104961</td>\n",
       "      <td>2.446307</td>\n",
       "      <td>1.111686</td>\n",
       "      <td>2.405894</td>\n",
       "      <td>1.111686</td>\n",
       "      <td>2.405894</td>\n",
       "      <td>1.114436</td>\n",
       "      <td>2.403211</td>\n",
       "      <td>1.113827</td>\n",
       "      <td>2.395075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>12000.0</td>\n",
       "      <td>3</td>\n",
       "      <td>12.99</td>\n",
       "      <td>404.27</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>65718</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>60000.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>770</td>\n",
       "      <td>13</td>\n",
       "      <td>19.22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>690.0</td>\n",
       "      <td>694.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27176.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1994</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>3257</td>\n",
       "      <td>0.224522</td>\n",
       "      <td>0.204005</td>\n",
       "      <td>1.125607</td>\n",
       "      <td>2.395431</td>\n",
       "      <td>1.085997</td>\n",
       "      <td>2.408741</td>\n",
       "      <td>0.984707</td>\n",
       "      <td>2.361605</td>\n",
       "      <td>1.141867</td>\n",
       "      <td>2.419815</td>\n",
       "      <td>1.133487</td>\n",
       "      <td>2.354374</td>\n",
       "      <td>1.100101</td>\n",
       "      <td>2.459716</td>\n",
       "      <td>1.119411</td>\n",
       "      <td>2.396658</td>\n",
       "      <td>1.136053</td>\n",
       "      <td>2.409156</td>\n",
       "      <td>1.011351</td>\n",
       "      <td>2.376224</td>\n",
       "      <td>1.124941</td>\n",
       "      <td>2.384061</td>\n",
       "      <td>1.111686</td>\n",
       "      <td>2.405894</td>\n",
       "      <td>1.111686</td>\n",
       "      <td>2.405894</td>\n",
       "      <td>1.114436</td>\n",
       "      <td>2.403211</td>\n",
       "      <td>0.923430</td>\n",
       "      <td>2.361914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>24000.0</td>\n",
       "      <td>3</td>\n",
       "      <td>9.99</td>\n",
       "      <td>774.30</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>209276</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>150000.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>8</td>\n",
       "      <td>5.68</td>\n",
       "      <td>0.0</td>\n",
       "      <td>690.0</td>\n",
       "      <td>694.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4334.0</td>\n",
       "      <td>68.8</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1983</td>\n",
       "      <td>18780</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2983</td>\n",
       "      <td>0.131210</td>\n",
       "      <td>0.128111</td>\n",
       "      <td>0.707941</td>\n",
       "      <td>1.635584</td>\n",
       "      <td>0.736477</td>\n",
       "      <td>1.592982</td>\n",
       "      <td>0.766491</td>\n",
       "      <td>1.620731</td>\n",
       "      <td>0.720818</td>\n",
       "      <td>1.621383</td>\n",
       "      <td>0.755658</td>\n",
       "      <td>1.569583</td>\n",
       "      <td>0.757800</td>\n",
       "      <td>1.549487</td>\n",
       "      <td>0.738697</td>\n",
       "      <td>1.625010</td>\n",
       "      <td>0.757368</td>\n",
       "      <td>1.606104</td>\n",
       "      <td>0.765499</td>\n",
       "      <td>1.622304</td>\n",
       "      <td>0.736884</td>\n",
       "      <td>1.643567</td>\n",
       "      <td>0.741124</td>\n",
       "      <td>1.603929</td>\n",
       "      <td>0.741124</td>\n",
       "      <td>1.603929</td>\n",
       "      <td>0.742958</td>\n",
       "      <td>1.602141</td>\n",
       "      <td>0.742552</td>\n",
       "      <td>1.596716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>16000.0</td>\n",
       "      <td>3</td>\n",
       "      <td>7.91</td>\n",
       "      <td>500.72</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>8198</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>50000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>76</td>\n",
       "      <td>8</td>\n",
       "      <td>38.95</td>\n",
       "      <td>0.0</td>\n",
       "      <td>710.0</td>\n",
       "      <td>714.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19023.0</td>\n",
       "      <td>60.8</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2011</td>\n",
       "      <td>16334</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3136</td>\n",
       "      <td>0.059838</td>\n",
       "      <td>0.083522</td>\n",
       "      <td>0.375202</td>\n",
       "      <td>0.798477</td>\n",
       "      <td>0.371135</td>\n",
       "      <td>0.810299</td>\n",
       "      <td>0.376013</td>\n",
       "      <td>0.793297</td>\n",
       "      <td>0.373832</td>\n",
       "      <td>0.789625</td>\n",
       "      <td>0.368325</td>\n",
       "      <td>0.815212</td>\n",
       "      <td>0.366700</td>\n",
       "      <td>0.819905</td>\n",
       "      <td>0.375204</td>\n",
       "      <td>0.784930</td>\n",
       "      <td>0.364666</td>\n",
       "      <td>0.813245</td>\n",
       "      <td>0.376035</td>\n",
       "      <td>0.793549</td>\n",
       "      <td>0.368003</td>\n",
       "      <td>0.809138</td>\n",
       "      <td>0.370562</td>\n",
       "      <td>0.801965</td>\n",
       "      <td>0.370562</td>\n",
       "      <td>0.801965</td>\n",
       "      <td>0.371479</td>\n",
       "      <td>0.801070</td>\n",
       "      <td>0.395135</td>\n",
       "      <td>0.846111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6000.0</td>\n",
       "      <td>3</td>\n",
       "      <td>10.49</td>\n",
       "      <td>194.99</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>115263</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>77000.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>106</td>\n",
       "      <td>38</td>\n",
       "      <td>17.27</td>\n",
       "      <td>0.0</td>\n",
       "      <td>660.0</td>\n",
       "      <td>664.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>220.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>49.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1996</td>\n",
       "      <td>18780</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3533</td>\n",
       "      <td>0.131210</td>\n",
       "      <td>0.109461</td>\n",
       "      <td>0.750404</td>\n",
       "      <td>1.596954</td>\n",
       "      <td>0.736477</td>\n",
       "      <td>1.592982</td>\n",
       "      <td>0.755989</td>\n",
       "      <td>1.626497</td>\n",
       "      <td>0.720818</td>\n",
       "      <td>1.621383</td>\n",
       "      <td>0.769944</td>\n",
       "      <td>1.605151</td>\n",
       "      <td>0.739618</td>\n",
       "      <td>1.580526</td>\n",
       "      <td>0.746274</td>\n",
       "      <td>1.597772</td>\n",
       "      <td>0.788374</td>\n",
       "      <td>1.610142</td>\n",
       "      <td>0.755638</td>\n",
       "      <td>1.627560</td>\n",
       "      <td>0.749961</td>\n",
       "      <td>1.589374</td>\n",
       "      <td>0.741124</td>\n",
       "      <td>1.603929</td>\n",
       "      <td>0.741124</td>\n",
       "      <td>1.603929</td>\n",
       "      <td>0.742958</td>\n",
       "      <td>1.602141</td>\n",
       "      <td>0.846155</td>\n",
       "      <td>1.753293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10375.0</td>\n",
       "      <td>5</td>\n",
       "      <td>15.61</td>\n",
       "      <td>250.16</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>74728</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>58000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>437</td>\n",
       "      <td>36</td>\n",
       "      <td>21.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>705.0</td>\n",
       "      <td>709.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>36609.0</td>\n",
       "      <td>61.1</td>\n",
       "      <td>33.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2002</td>\n",
       "      <td>18780</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2526</td>\n",
       "      <td>0.304227</td>\n",
       "      <td>0.279444</td>\n",
       "      <td>1.500809</td>\n",
       "      <td>3.193909</td>\n",
       "      <td>1.502905</td>\n",
       "      <td>3.185919</td>\n",
       "      <td>1.511979</td>\n",
       "      <td>3.252993</td>\n",
       "      <td>1.494754</td>\n",
       "      <td>3.218213</td>\n",
       "      <td>1.473298</td>\n",
       "      <td>3.260850</td>\n",
       "      <td>1.479236</td>\n",
       "      <td>3.161051</td>\n",
       "      <td>1.492548</td>\n",
       "      <td>3.195544</td>\n",
       "      <td>1.497336</td>\n",
       "      <td>3.234727</td>\n",
       "      <td>1.511277</td>\n",
       "      <td>3.255120</td>\n",
       "      <td>1.496655</td>\n",
       "      <td>3.146687</td>\n",
       "      <td>1.482248</td>\n",
       "      <td>3.207858</td>\n",
       "      <td>1.482248</td>\n",
       "      <td>3.207858</td>\n",
       "      <td>1.485915</td>\n",
       "      <td>3.204282</td>\n",
       "      <td>1.485103</td>\n",
       "      <td>3.193433</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   loanAmnt  term  interestRate  installment  grade  subGrade  \\\n",
       "0   35000.0     5         19.52       917.97      5        21   \n",
       "1   18000.0     5         18.49       461.90      4        16   \n",
       "2   12000.0     5         16.99       298.17      4        17   \n",
       "3    2050.0     3          7.69        63.95      1         3   \n",
       "4   11500.0     3         14.98       398.54      3        12   \n",
       "5   12000.0     3         12.99       404.27      3        11   \n",
       "6   24000.0     3          9.99       774.30      2         7   \n",
       "7   16000.0     3          7.91       500.72      1         4   \n",
       "8    6000.0     3         10.49       194.99      2         6   \n",
       "9   10375.0     5         15.61       250.16      4        15   \n",
       "\n",
       "   employmentTitle  employmentLength  homeOwnership  annualIncome  \\\n",
       "0           161280                 2              2      110000.0   \n",
       "1            89538                 5              0       46000.0   \n",
       "2           159367                 8              0       74000.0   \n",
       "3            59830                 9              0       35000.0   \n",
       "4            85242                 1              1       30000.0   \n",
       "5            65718                 5              2       60000.0   \n",
       "6           209276                10              0      150000.0   \n",
       "7             8198                 2              1       50000.0   \n",
       "8           115263                 2              0       77000.0   \n",
       "9            74728                 9              0       58000.0   \n",
       "\n",
       "   verificationStatus  isDefault  purpose  postCode  regionCode    dti  \\\n",
       "0                   2          1        1        43          32  17.05   \n",
       "1                   2          0        0        64          18  27.83   \n",
       "2                   2          0        0       265          14  22.77   \n",
       "3                   0          0        0       465          14  17.49   \n",
       "4                   2          0        0         3           4  32.60   \n",
       "5                   1          1        0       770          13  19.22   \n",
       "6                   1          0        2        40           8   5.68   \n",
       "7                   0          0        4        76           8  38.95   \n",
       "8                   1          0        2       106          38  17.27   \n",
       "9                   0          0        2       437          36  21.02   \n",
       "\n",
       "   delinquency_2years  ficoRangeLow  ficoRangeHigh  openAcc  pubRec  \\\n",
       "0                 0.0         730.0          734.0      7.0     0.0   \n",
       "1                 0.0         700.0          704.0     13.0     0.0   \n",
       "2                 0.0         675.0          679.0     11.0     0.0   \n",
       "3                 0.0         755.0          759.0     12.0     0.0   \n",
       "4                 0.0         665.0          669.0      8.0     1.0   \n",
       "5                 0.0         690.0          694.0     15.0     0.0   \n",
       "6                 0.0         690.0          694.0      7.0     0.0   \n",
       "7                 0.0         710.0          714.0      9.0     0.0   \n",
       "8                 0.0         660.0          664.0     16.0     1.0   \n",
       "9                 0.0         705.0          709.0     16.0     0.0   \n",
       "\n",
       "   pubRecBankruptcies  revolBal  revolUtil  totalAcc  initialListStatus  \\\n",
       "0                 0.0   24178.0       48.9      27.0                  0   \n",
       "1                 0.0   15096.0       38.9      18.0                  1   \n",
       "2                 0.0    4606.0       51.8      27.0                  0   \n",
       "3                 0.0    3111.0        8.5      23.0                  0   \n",
       "4                 1.0   14021.0       59.7      33.0                  1   \n",
       "5                 0.0   27176.0       46.0      21.0                  1   \n",
       "6                 0.0    4334.0       68.8      25.0                  0   \n",
       "7                 0.0   19023.0       60.8      11.0                  0   \n",
       "8                 1.0     220.0        3.6      49.0                  0   \n",
       "9                 0.0   36609.0       61.1      33.0                  0   \n",
       "\n",
       "   applicationType  earliesCreditLine  title  policyCode   n0   n1    n2  \\\n",
       "0                0               2001      1         1.0  0.0  2.0   2.0   \n",
       "1                0               2002   5768         1.0  0.0  3.0   5.0   \n",
       "2                0               2006      0         1.0  0.0  0.0   3.0   \n",
       "3                0               2006      0         1.0  0.0  1.0   3.0   \n",
       "4                0               1994      0         1.0  0.0  4.0   4.0   \n",
       "5                0               1994      0         1.0  0.0  7.0  13.0   \n",
       "6                0               1983  18780         1.0  1.0  1.0   3.0   \n",
       "7                0               2011  16334         1.0  0.0  4.0   5.0   \n",
       "8                0               1996  18780         1.0  0.0  1.0   4.0   \n",
       "9                0               2002  18780         1.0  0.0  3.0   4.0   \n",
       "\n",
       "     n3    n4    n5    n6    n7    n8    n9   n10  n11  n12  n13  n14  \\\n",
       "0   2.0   4.0   9.0   8.0   4.0  12.0   2.0   7.0  0.0  0.0  0.0  2.0   \n",
       "1   5.0  10.0   7.0   7.0   7.0  13.0   5.0  13.0  0.0  0.0  0.0  2.0   \n",
       "2   3.0   0.0   0.0  21.0   4.0   5.0   3.0  11.0  0.0  0.0  0.0  4.0   \n",
       "3   3.0   7.0  11.0   3.0  10.0  18.0   3.0  12.0  0.0  0.0  0.0  3.0   \n",
       "4   4.0   4.0  16.0  10.0   5.0  21.0   4.0   8.0  0.0  0.0  0.0  2.0   \n",
       "5  13.0   7.0   7.0   2.0  13.0  17.0  11.0  15.0  0.0  0.0  0.0  6.0   \n",
       "6   3.0   2.0   7.0   7.0   6.0  17.0   3.0   7.0  0.0  0.0  0.0  2.0   \n",
       "7   5.0   4.0   6.0   2.0   7.0   9.0   5.0   9.0  0.0  0.0  0.0  1.0   \n",
       "8   4.0   2.0  11.0  14.0  13.0  32.0   4.0  15.0  0.0  0.0  0.0  0.0   \n",
       "9   4.0   5.0   6.0  14.0  13.0  14.0   4.0  16.0  0.0  0.0  0.0  2.0   \n",
       "\n",
       "   issueDateDT  grade_target_mean  subGrade_target_mean  grade_to_mean_n0  \\\n",
       "0         2587           0.386234              0.380444          1.876011   \n",
       "1         1888           0.304227              0.298190          1.500809   \n",
       "2         3044           0.304227              0.302541          1.500809   \n",
       "3         2679           0.059838              0.065532          0.375202   \n",
       "4         2406           0.224522              0.224686          1.125607   \n",
       "5         3257           0.224522              0.204005          1.125607   \n",
       "6         2983           0.131210              0.128111          0.707941   \n",
       "7         3136           0.059838              0.083522          0.375202   \n",
       "8         3533           0.131210              0.109461          0.750404   \n",
       "9         2526           0.304227              0.279444          1.500809   \n",
       "\n",
       "   grade_to_std_n0  grade_to_mean_n1  grade_to_std_n1  grade_to_mean_n2  \\\n",
       "0         3.992386          1.874620         4.053876          1.942294   \n",
       "1         3.193909          1.502905         3.185919          1.504054   \n",
       "2         3.193909          1.360761         2.998190          1.532981   \n",
       "3         0.798477          0.368239         0.796491          0.383245   \n",
       "4         2.395431          1.113406         2.430896          1.133984   \n",
       "5         2.395431          1.085997         2.408741          0.984707   \n",
       "6         1.635584          0.736477         1.592982          0.766491   \n",
       "7         0.798477          0.371135         0.810299          0.376013   \n",
       "8         1.596954          0.736477         1.592982          0.755989   \n",
       "9         3.193909          1.502905         3.185919          1.511979   \n",
       "\n",
       "   grade_to_std_n2  grade_to_mean_n4  grade_to_std_n4  grade_to_mean_n5  \\\n",
       "0         4.023418          1.869160         3.948124          1.897562   \n",
       "1         3.173189          1.567352         3.204484          1.511316   \n",
       "2         3.241462          1.273891         3.071276          1.162371   \n",
       "3         0.810366          0.380622         0.806605          0.384972   \n",
       "4         2.439745          1.121496         2.368874          1.197930   \n",
       "5         2.361605          1.141867         2.419815          1.133487   \n",
       "6         1.620731          0.720818         1.621383          0.755658   \n",
       "7         0.793297          0.373832         0.789625          0.368325   \n",
       "8         1.626497          0.720818         1.621383          0.769944   \n",
       "9         3.252993          1.494754         3.218213          1.473298   \n",
       "\n",
       "   grade_to_std_n5  grade_to_mean_n6  grade_to_std_n6  grade_to_mean_n7  \\\n",
       "0         4.055665          1.865760         4.017884          1.840872   \n",
       "1         3.139166          1.515599         3.098975          1.500817   \n",
       "2         3.176718          1.480241         3.125317          1.472698   \n",
       "3         0.802575          0.368526         0.819126          0.369865   \n",
       "4         2.401168          1.120956         2.388727          1.106851   \n",
       "5         2.354374          1.100101         2.459716          1.119411   \n",
       "6         1.569583          0.757800         1.549487          0.738697   \n",
       "7         0.815212          0.366700         0.819905          0.375204   \n",
       "8         1.605151          0.739618         1.580526          0.746274   \n",
       "9         3.260850          1.479236         3.161051          1.492548   \n",
       "\n",
       "   grade_to_std_n7  grade_to_mean_n8  grade_to_std_n8  grade_to_mean_n9  \\\n",
       "0         4.074681          1.851544         4.040923          1.938318   \n",
       "1         3.139721          1.517874         3.086106          1.504140   \n",
       "2         3.259745          1.406712         3.254085          1.530998   \n",
       "3         0.798404          0.377964         0.799464          0.382750   \n",
       "4         2.450979          1.144817         2.403154          1.133458   \n",
       "5         2.396658          1.136053         2.409156          1.011351   \n",
       "6         1.625010          0.757368         1.606104          0.765499   \n",
       "7         0.784930          0.364666         0.813245          0.376035   \n",
       "8         1.597772          0.788374         1.610142          0.755638   \n",
       "9         3.195544          1.497336         3.234727          1.511277   \n",
       "\n",
       "   grade_to_std_n9  grade_to_mean_n10  grade_to_std_n10  grade_to_mean_n11  \\\n",
       "0         4.024912           1.842210          4.108917           1.852810   \n",
       "1         3.174194           1.484104          3.173687           1.482248   \n",
       "2         3.244609           1.504230          3.089208           1.482248   \n",
       "3         0.811152           0.370128          0.799459           0.370562   \n",
       "4         2.441340           1.104961          2.446307           1.111686   \n",
       "5         2.376224           1.124941          2.384061           1.111686   \n",
       "6         1.622304           0.736884          1.643567           0.741124   \n",
       "7         0.793549           0.368003          0.809138           0.370562   \n",
       "8         1.627560           0.749961          1.589374           0.741124   \n",
       "9         3.255120           1.496655          3.146687           1.482248   \n",
       "\n",
       "   grade_to_std_n11  grade_to_mean_n12  grade_to_std_n12  grade_to_mean_n13  \\\n",
       "0          4.009823           1.852810          4.009823           1.857394   \n",
       "1          3.207858           1.482248          3.207858           1.485915   \n",
       "2          3.207858           1.482248          3.207858           1.485915   \n",
       "3          0.801965           0.370562          0.801965           0.371479   \n",
       "4          2.405894           1.111686          2.405894           1.114436   \n",
       "5          2.405894           1.111686          2.405894           1.114436   \n",
       "6          1.603929           0.741124          1.603929           0.742958   \n",
       "7          0.801965           0.370562          0.801965           0.371479   \n",
       "8          1.603929           0.741124          1.603929           0.742958   \n",
       "9          3.207858           1.482248          3.207858           1.485915   \n",
       "\n",
       "   grade_to_std_n13  grade_to_mean_n14  grade_to_std_n14  \n",
       "0          4.005352           1.856379          3.991791  \n",
       "1          3.204282           1.485103          3.193433  \n",
       "2          3.204282           1.315111          3.146801  \n",
       "3          0.801070           0.344287          0.793451  \n",
       "4          2.403211           1.113827          2.395075  \n",
       "5          2.403211           0.923430          2.361914  \n",
       "6          1.602141           0.742552          1.596716  \n",
       "7          0.801070           0.395135          0.846111  \n",
       "8          1.602141           0.846155          1.753293  \n",
       "9          3.204282           1.485103          3.193433  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd621929-0a30-4dfb-93e8-5f6bb535b67e",
   "metadata": {},
   "source": [
    "### Label and Features \n",
    "Suppose the Spark Dataframe of this training dataset only contains numerical features. Here we use `params['label']` column value as label and other columns as features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0ee3a777-d142-411d-bc4c-8053cc1a1065",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_config(path):\n",
    "    params = dict()\n",
    "    with open(path, 'r') as stream:\n",
    "        params = yaml.load(stream, Loader=yaml.FullLoader)\n",
    "    return params\n",
    "\n",
    "params = load_config('../conf/spark_lgbm_dev.yaml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b113437c-395c-4df4-a08d-74fe245a5f55",
   "metadata": {},
   "outputs": [],
   "source": [
    "label = params['label']\n",
    "feature_cols = [x for x in train_data.columns if x not in [label]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9b90f5d9-997a-4d87-8f45-3d8d66d216fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['loanAmnt',\n",
       " 'term',\n",
       " 'interestRate',\n",
       " 'installment',\n",
       " 'grade',\n",
       " 'subGrade',\n",
       " 'employmentTitle',\n",
       " 'employmentLength',\n",
       " 'homeOwnership',\n",
       " 'annualIncome',\n",
       " 'verificationStatus',\n",
       " 'purpose',\n",
       " 'postCode',\n",
       " 'regionCode',\n",
       " 'dti',\n",
       " 'delinquency_2years',\n",
       " 'ficoRangeLow',\n",
       " 'ficoRangeHigh',\n",
       " 'openAcc',\n",
       " 'pubRec',\n",
       " 'pubRecBankruptcies',\n",
       " 'revolBal',\n",
       " 'revolUtil',\n",
       " 'totalAcc',\n",
       " 'initialListStatus',\n",
       " 'applicationType',\n",
       " 'earliesCreditLine',\n",
       " 'title',\n",
       " 'policyCode',\n",
       " 'n0',\n",
       " 'n1',\n",
       " 'n2',\n",
       " 'n3',\n",
       " 'n4',\n",
       " 'n5',\n",
       " 'n6',\n",
       " 'n7',\n",
       " 'n8',\n",
       " 'n9',\n",
       " 'n10',\n",
       " 'n11',\n",
       " 'n12',\n",
       " 'n13',\n",
       " 'n14',\n",
       " 'issueDateDT',\n",
       " 'grade_target_mean',\n",
       " 'subGrade_target_mean',\n",
       " 'grade_to_mean_n0',\n",
       " 'grade_to_std_n0',\n",
       " 'grade_to_mean_n1',\n",
       " 'grade_to_std_n1',\n",
       " 'grade_to_mean_n2',\n",
       " 'grade_to_std_n2',\n",
       " 'grade_to_mean_n4',\n",
       " 'grade_to_std_n4',\n",
       " 'grade_to_mean_n5',\n",
       " 'grade_to_std_n5',\n",
       " 'grade_to_mean_n6',\n",
       " 'grade_to_std_n6',\n",
       " 'grade_to_mean_n7',\n",
       " 'grade_to_std_n7',\n",
       " 'grade_to_mean_n8',\n",
       " 'grade_to_std_n8',\n",
       " 'grade_to_mean_n9',\n",
       " 'grade_to_std_n9',\n",
       " 'grade_to_mean_n10',\n",
       " 'grade_to_std_n10',\n",
       " 'grade_to_mean_n11',\n",
       " 'grade_to_std_n11',\n",
       " 'grade_to_mean_n12',\n",
       " 'grade_to_std_n12',\n",
       " 'grade_to_mean_n13',\n",
       " 'grade_to_std_n13',\n",
       " 'grade_to_mean_n14',\n",
       " 'grade_to_std_n14']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6232a73-829b-4a87-82f8-2cedf9c13d7c",
   "metadata": {},
   "source": [
    "### Train and evaluation\n",
    "In this section, we will use LightGBM to build our binary classification model. The meaning of model hyper parameters can be referred to:\n",
    " * https://lightgbm.readthedocs.io/en/latest/Parameters.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8d69d5b7-cf6a-4ee4-b449-40f15a5d6fb9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.749044\tvalid_1's auc: 0.72963\n",
      "[400]\ttraining's auc: 0.764902\tvalid_1's auc: 0.730119\n",
      "[600]\ttraining's auc: 0.778805\tvalid_1's auc: 0.729919\n",
      "Early stopping, best iteration is:\n",
      "[408]\ttraining's auc: 0.765464\tvalid_1's auc: 0.730148\n",
      "Feature Importance:\n",
      " [('subGrade', 55731.91763496399), ('subGrade_target_mean', 25544.73084115982), ('issueDateDT', 23477.302619218826), ('grade_to_mean_n4', 20431.835428237915), ('grade_to_mean_n7', 17742.464669704437), ('term', 13944.866245031357), ('grade_to_std_n4', 11223.571625232697), ('dti', 11088.541720628738), ('annualIncome', 10994.476479291916), ('homeOwnership', 9251.973315000534), ('revolBal', 8729.287752866745), ('employmentTitle', 8060.54189157486), ('loanAmnt', 7468.738019227982), ('installment', 6830.989168405533), ('regionCode', 6447.618449687958), ('ficoRangeLow', 5539.447093009949), ('revolUtil', 5221.779930591583), ('earliesCreditLine', 4704.3636956214905), ('totalAcc', 4580.0606944561005), ('grade_to_mean_n1', 4530.397274971008), ('postCode', 4296.541624069214), ('grade_to_mean_n10', 4279.063055038452), ('n2', 3860.0260243415833), ('interestRate', 3577.4857523441315), ('n14', 3274.0861802101135), ('grade_to_std_n7', 3053.215183496475), ('grade_to_std_n8', 2641.1590645313263), ('grade_to_mean_n5', 2536.729082584381), ('n6', 2530.2121226787567), ('employmentLength', 2478.785659790039)]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "train, valid = train_test_split(train_data, test_size=0.2, random_state=1)\n",
    "X_train, y_train = train[feature_cols], train[label]\n",
    "X_valid, y_valid = valid[feature_cols], valid[label]\n",
    "train_matrix = lgb.Dataset(X_train, label=y_train)\n",
    "valid_matrix = lgb.Dataset(X_valid, label=y_valid)\n",
    "\n",
    "params = {\n",
    "    'boosting_type': 'gbdt',\n",
    "    'objective': 'binary',\n",
    "    'metric': 'auc',\n",
    "    'min_child_weight': 5,\n",
    "    'num_leaves': 2 ** 5,\n",
    "    'lambda_l2': 10,\n",
    "    'feature_fraction': 0.8,\n",
    "    'bagging_fraction': 0.8,\n",
    "    'bagging_freq': 4,\n",
    "    'learning_rate': 0.1,\n",
    "    'seed': 2022,\n",
    "    'nthread': 28,\n",
    "    'n_jobs':24,\n",
    "    'silent': True,\n",
    "    'verbose': -1,\n",
    "\n",
    "}\n",
    "\n",
    "model = lgb.train(params, train_matrix, 50000, valid_sets=[train_matrix, valid_matrix], verbose_eval=200, early_stopping_rounds=200)\n",
    "print(\"Feature Importance:\\n\", list(sorted(zip(X_train.columns, model.feature_importance(\"gain\")), key=lambda x: x[1], reverse=True))[:30])     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "549c30d1-82c5-42a8-aee4-ff2bf172d5af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train dataset auc: 0.7654641469179302\n",
      "validation dataset auc: 0.7301476655170198\n"
     ]
    }
   ],
   "source": [
    "pred_train = model.predict(X_train, num_iteration=model.best_iteration)     \n",
    "pred_valid = model.predict(X_valid, num_iteration=model.best_iteration)            \n",
    "\n",
    "print(\"train dataset auc:\", roc_auc_score(y_train, pred_train))\n",
    "print(\"validation dataset auc:\", roc_auc_score(y_valid, pred_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cd29a00-7595-4165-8064-f7f262601963",
   "metadata": {},
   "source": [
    "### Hyper Parameters Tuning\n",
    "In this section, we will use `BayesianOptimization` to tune the hyper parameters of the lightgbm model. In this demo, we fix the `learning_rate` and `n_estimators` to search other hyper parameter combinations. According to its [Github issue](https://github.com/fmfn/BayesianOptimization/issues/300), we should make sure the scipy version equals to `1.7`. Please refer to `../requirements.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "49897d14-a088-41c8-97c4-f59977789f97",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train_data[feature_cols]\n",
    "y_train = train_data[label]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1fd07203-10a3-48fc-91a6-62c6f0812e2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "def rf_cv_lgb(X_train,\n",
    "              y_train,\n",
    "              num_leaves, \n",
    "              max_depth,\n",
    "              bagging_fraction, \n",
    "              feature_fraction, \n",
    "              bagging_freq, \n",
    "              min_data_in_leaf, \n",
    "              min_child_weight, \n",
    "              min_split_gain, \n",
    "              reg_lambda, \n",
    "              reg_alpha,\n",
    "              cv=5,\n",
    "              scoreing='roc_auc',\n",
    "              **kwargs):\n",
    "    model_lgb = lgb.LGBMClassifier(\n",
    "        boosting_type='gbdt', \n",
    "        objective='binary', \n",
    "        metric='auc',             \n",
    "        learning_rate=0.1, \n",
    "        n_estimators=500,\n",
    "        num_leaves=int(num_leaves), \n",
    "        max_depth=int(max_depth),          \n",
    "        bagging_fraction=round(bagging_fraction, 2), \n",
    "        feature_fraction=round(feature_fraction, 2),                           \n",
    "        bagging_freq=int(bagging_freq), \n",
    "        min_data_in_leaf=int(min_data_in_leaf),\n",
    "        min_child_weight=min_child_weight, \n",
    "        min_split_gain=min_split_gain,\n",
    "        reg_lambda=reg_lambda, \n",
    "        reg_alpha=reg_alpha,\n",
    "        n_jobs=8)\n",
    "    \n",
    "    val = cross_val_score(model_lgb, X_train, y_train, cv=cv, scoring='roc_auc').mean()\n",
    "    \n",
    "    return val\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "from bayes_opt import BayesianOptimization\n",
    "bayes_lgb = BayesianOptimization(\n",
    "    partial(rf_cv_lgb, X_train=X_train, y_train=y_train), \n",
    "    {\n",
    "        'num_leaves':(10, 200),\n",
    "        'max_depth':(3, 20),\n",
    "        'bagging_fraction':(0.5, 1.0),\n",
    "        'feature_fraction':(0.5, 1.0),\n",
    "        'bagging_freq':(0, 100),\n",
    "        'min_data_in_leaf':(10,100),\n",
    "        'min_child_weight':(0, 10),\n",
    "        'min_split_gain':(0.0, 1.0),\n",
    "        'reg_alpha':(0.0, 10),\n",
    "        'reg_lambda':(0.0, 10),\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "04715dfd-d8f4-4ac2-8204-91dc57f39aa8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   | baggin... | baggin... | featur... | max_depth | min_ch... | min_da... | min_sp... | num_le... | reg_alpha | reg_la... |\n",
      "-------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "[LightGBM] [Warning] feature_fraction is set=0.74, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.74\n",
      "[LightGBM] [Warning] bagging_freq is set=41, subsample_freq=0 will be ignored. Current value: bagging_freq=41\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=42, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=42\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.74, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.74\n",
      "[LightGBM] [Warning] bagging_freq is set=41, subsample_freq=0 will be ignored. Current value: bagging_freq=41\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=42, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=42\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.74, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.74\n",
      "[LightGBM] [Warning] bagging_freq is set=41, subsample_freq=0 will be ignored. Current value: bagging_freq=41\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=42, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=42\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.74, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.74\n",
      "[LightGBM] [Warning] bagging_freq is set=41, subsample_freq=0 will be ignored. Current value: bagging_freq=41\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=42, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=42\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.74, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.74\n",
      "[LightGBM] [Warning] bagging_freq is set=41, subsample_freq=0 will be ignored. Current value: bagging_freq=41\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=42, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=42\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 0.7311  \u001b[0m | \u001b[0m 0.7511  \u001b[0m | \u001b[0m 41.04   \u001b[0m | \u001b[0m 0.7437  \u001b[0m | \u001b[0m 12.45   \u001b[0m | \u001b[0m 7.707   \u001b[0m | \u001b[0m 42.82   \u001b[0m | \u001b[0m 0.7638  \u001b[0m | \u001b[0m 70.27   \u001b[0m | \u001b[0m 8.191   \u001b[0m | \u001b[0m 0.4402  \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=0.94, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.94\n",
      "[LightGBM] [Warning] bagging_freq is set=92, subsample_freq=0 will be ignored. Current value: bagging_freq=92\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=89, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=89\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.65, subsample=1.0 will be ignored. Current value: bagging_fraction=0.65\n",
      "[LightGBM] [Warning] feature_fraction is set=0.94, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.94\n",
      "[LightGBM] [Warning] bagging_freq is set=92, subsample_freq=0 will be ignored. Current value: bagging_freq=92\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=89, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=89\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.65, subsample=1.0 will be ignored. Current value: bagging_fraction=0.65\n",
      "[LightGBM] [Warning] feature_fraction is set=0.94, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.94\n",
      "[LightGBM] [Warning] bagging_freq is set=92, subsample_freq=0 will be ignored. Current value: bagging_freq=92\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=89, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=89\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.65, subsample=1.0 will be ignored. Current value: bagging_fraction=0.65\n",
      "[LightGBM] [Warning] feature_fraction is set=0.94, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.94\n",
      "[LightGBM] [Warning] bagging_freq is set=92, subsample_freq=0 will be ignored. Current value: bagging_freq=92\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=89, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=89\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.65, subsample=1.0 will be ignored. Current value: bagging_fraction=0.65\n",
      "[LightGBM] [Warning] feature_fraction is set=0.94, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.94\n",
      "[LightGBM] [Warning] bagging_freq is set=92, subsample_freq=0 will be ignored. Current value: bagging_freq=92\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=89, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=89\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.65, subsample=1.0 will be ignored. Current value: bagging_fraction=0.65\n",
      "| \u001b[0m 2       \u001b[0m | \u001b[0m 0.7299  \u001b[0m | \u001b[0m 0.6489  \u001b[0m | \u001b[0m 92.2    \u001b[0m | \u001b[0m 0.9367  \u001b[0m | \u001b[0m 19.24   \u001b[0m | \u001b[0m 4.542   \u001b[0m | \u001b[0m 89.16   \u001b[0m | \u001b[0m 0.505   \u001b[0m | \u001b[0m 24.56   \u001b[0m | \u001b[0m 0.1941  \u001b[0m | \u001b[0m 2.123   \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] bagging_freq is set=36, subsample_freq=0 will be ignored. Current value: bagging_freq=36\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] bagging_freq is set=36, subsample_freq=0 will be ignored. Current value: bagging_freq=36\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] bagging_freq is set=36, subsample_freq=0 will be ignored. Current value: bagging_freq=36\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] bagging_freq is set=36, subsample_freq=0 will be ignored. Current value: bagging_freq=36\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] bagging_freq is set=36, subsample_freq=0 will be ignored. Current value: bagging_freq=36\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m 0.7266  \u001b[0m | \u001b[0m 0.6861  \u001b[0m | \u001b[0m 36.82   \u001b[0m | \u001b[0m 0.8071  \u001b[0m | \u001b[0m 15.89   \u001b[0m | \u001b[0m 2.367   \u001b[0m | \u001b[0m 77.38   \u001b[0m | \u001b[0m 0.09882 \u001b[0m | \u001b[0m 107.3   \u001b[0m | \u001b[0m 2.223   \u001b[0m | \u001b[0m 4.002   \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=0.96, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.96\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.6, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6\n",
      "[LightGBM] [Warning] feature_fraction is set=0.96, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.96\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.6, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6\n",
      "[LightGBM] [Warning] feature_fraction is set=0.96, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.96\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.6, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6\n",
      "[LightGBM] [Warning] feature_fraction is set=0.96, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.96\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.6, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6\n",
      "[LightGBM] [Warning] feature_fraction is set=0.96, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.96\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.6, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6\n",
      "| \u001b[0m 4       \u001b[0m | \u001b[0m 0.7277  \u001b[0m | \u001b[0m 0.6034  \u001b[0m | \u001b[0m 8.938   \u001b[0m | \u001b[0m 0.9555  \u001b[0m | \u001b[0m 13.73   \u001b[0m | \u001b[0m 8.7     \u001b[0m | \u001b[0m 48.37   \u001b[0m | \u001b[0m 0.9722  \u001b[0m | \u001b[0m 117.8   \u001b[0m | \u001b[0m 9.964   \u001b[0m | \u001b[0m 7.096   \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=0.59, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.59\n",
      "[LightGBM] [Warning] bagging_freq is set=24, subsample_freq=0 will be ignored. Current value: bagging_freq=24\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "[LightGBM] [Warning] feature_fraction is set=0.59, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.59\n",
      "[LightGBM] [Warning] bagging_freq is set=24, subsample_freq=0 will be ignored. Current value: bagging_freq=24\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "[LightGBM] [Warning] feature_fraction is set=0.59, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.59\n",
      "[LightGBM] [Warning] bagging_freq is set=24, subsample_freq=0 will be ignored. Current value: bagging_freq=24\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "[LightGBM] [Warning] feature_fraction is set=0.59, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.59\n",
      "[LightGBM] [Warning] bagging_freq is set=24, subsample_freq=0 will be ignored. Current value: bagging_freq=24\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "[LightGBM] [Warning] feature_fraction is set=0.59, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.59\n",
      "[LightGBM] [Warning] bagging_freq is set=24, subsample_freq=0 will be ignored. Current value: bagging_freq=24\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m 0.7227  \u001b[0m | \u001b[0m 0.5023  \u001b[0m | \u001b[0m 24.03   \u001b[0m | \u001b[0m 0.5859  \u001b[0m | \u001b[0m 17.24   \u001b[0m | \u001b[0m 0.7628  \u001b[0m | \u001b[0m 62.42   \u001b[0m | \u001b[0m 0.1768  \u001b[0m | \u001b[0m 150.4   \u001b[0m | \u001b[0m 4.503   \u001b[0m | \u001b[0m 9.362   \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] bagging_freq is set=52, subsample_freq=0 will be ignored. Current value: bagging_freq=52\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] bagging_freq is set=52, subsample_freq=0 will be ignored. Current value: bagging_freq=52\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] bagging_freq is set=52, subsample_freq=0 will be ignored. Current value: bagging_freq=52\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] bagging_freq is set=52, subsample_freq=0 will be ignored. Current value: bagging_freq=52\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] bagging_freq is set=52, subsample_freq=0 will be ignored. Current value: bagging_freq=52\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=48, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=48\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.69, subsample=1.0 will be ignored. Current value: bagging_fraction=0.69\n",
      "| \u001b[95m 6       \u001b[0m | \u001b[95m 0.7314  \u001b[0m | \u001b[95m 0.6915  \u001b[0m | \u001b[95m 52.52   \u001b[0m | \u001b[95m 0.543   \u001b[0m | \u001b[95m 13.84   \u001b[0m | \u001b[95m 8.503   \u001b[0m | \u001b[95m 48.75   \u001b[0m | \u001b[95m 0.09221 \u001b[0m | \u001b[95m 29.83   \u001b[0m | \u001b[95m 4.825   \u001b[0m | \u001b[95m 6.507   \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=0.64, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.64\n",
      "[LightGBM] [Warning] bagging_freq is set=54, subsample_freq=0 will be ignored. Current value: bagging_freq=54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=51, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=51\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.84, subsample=1.0 will be ignored. Current value: bagging_fraction=0.84\n",
      "[LightGBM] [Warning] feature_fraction is set=0.64, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.64\n",
      "[LightGBM] [Warning] bagging_freq is set=54, subsample_freq=0 will be ignored. Current value: bagging_freq=54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=51, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=51\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.84, subsample=1.0 will be ignored. Current value: bagging_fraction=0.84\n",
      "[LightGBM] [Warning] feature_fraction is set=0.64, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.64\n",
      "[LightGBM] [Warning] bagging_freq is set=54, subsample_freq=0 will be ignored. Current value: bagging_freq=54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=51, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=51\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.84, subsample=1.0 will be ignored. Current value: bagging_fraction=0.84\n",
      "[LightGBM] [Warning] feature_fraction is set=0.64, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.64\n",
      "[LightGBM] [Warning] bagging_freq is set=54, subsample_freq=0 will be ignored. Current value: bagging_freq=54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=51, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=51\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.84, subsample=1.0 will be ignored. Current value: bagging_fraction=0.84\n",
      "[LightGBM] [Warning] feature_fraction is set=0.64, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.64\n",
      "[LightGBM] [Warning] bagging_freq is set=54, subsample_freq=0 will be ignored. Current value: bagging_freq=54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=51, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=51\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.84, subsample=1.0 will be ignored. Current value: bagging_fraction=0.84\n",
      "| \u001b[95m 7       \u001b[0m | \u001b[95m 0.7318  \u001b[0m | \u001b[95m 0.8417  \u001b[0m | \u001b[95m 54.59   \u001b[0m | \u001b[95m 0.6386  \u001b[0m | \u001b[95m 12.05   \u001b[0m | \u001b[95m 9.392   \u001b[0m | \u001b[95m 51.3    \u001b[0m | \u001b[95m 0.4774  \u001b[0m | \u001b[95m 34.64   \u001b[0m | \u001b[95m 6.352   \u001b[0m | \u001b[95m 8.733   \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=100, subsample_freq=0 will be ignored. Current value: bagging_freq=100\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=100, subsample_freq=0 will be ignored. Current value: bagging_freq=100\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=100, subsample_freq=0 will be ignored. Current value: bagging_freq=100\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=100, subsample_freq=0 will be ignored. Current value: bagging_freq=100\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=100, subsample_freq=0 will be ignored. Current value: bagging_freq=100\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "| \u001b[0m 8       \u001b[0m | \u001b[0m 0.7299  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 100.0   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 3.0     \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 55.5    \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 10.0    \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=100, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=100\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "| \u001b[0m 9       \u001b[0m | \u001b[0m 0.7299  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 12.89   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 3.0     \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 100.0   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 24.6    \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 10.0    \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=18, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=18\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=18, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=18\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=18, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=18\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=18, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=18\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=18, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=18\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "| \u001b[0m 10      \u001b[0m | \u001b[0m 0.7299  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 3.0     \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 18.89   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 42.77   \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 10.0    \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=0.85, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.85\n",
      "[LightGBM] [Warning] bagging_freq is set=93, subsample_freq=0 will be ignored. Current value: bagging_freq=93\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=66, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=66\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.85, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.85\n",
      "[LightGBM] [Warning] bagging_freq is set=93, subsample_freq=0 will be ignored. Current value: bagging_freq=93\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=66, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=66\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.85, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.85\n",
      "[LightGBM] [Warning] bagging_freq is set=93, subsample_freq=0 will be ignored. Current value: bagging_freq=93\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=66, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=66\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.85, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.85\n",
      "[LightGBM] [Warning] bagging_freq is set=93, subsample_freq=0 will be ignored. Current value: bagging_freq=93\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=66, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=66\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.85, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.85\n",
      "[LightGBM] [Warning] bagging_freq is set=93, subsample_freq=0 will be ignored. Current value: bagging_freq=93\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=66, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=66\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "| \u001b[0m 11      \u001b[0m | \u001b[0m 0.7292  \u001b[0m | \u001b[0m 0.5906  \u001b[0m | \u001b[0m 93.47   \u001b[0m | \u001b[0m 0.8501  \u001b[0m | \u001b[0m 3.586   \u001b[0m | \u001b[0m 0.4441  \u001b[0m | \u001b[0m 66.82   \u001b[0m | \u001b[0m 0.9214  \u001b[0m | \u001b[0m 75.5    \u001b[0m | \u001b[0m 9.228   \u001b[0m | \u001b[0m 1.427   \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=0.72, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.72\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76\n",
      "[LightGBM] [Warning] feature_fraction is set=0.72, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.72\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76\n",
      "[LightGBM] [Warning] feature_fraction is set=0.72, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.72\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76\n",
      "[LightGBM] [Warning] feature_fraction is set=0.72, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.72\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76\n",
      "[LightGBM] [Warning] feature_fraction is set=0.72, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.72\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76\n",
      "| \u001b[0m 12      \u001b[0m | \u001b[0m 0.7306  \u001b[0m | \u001b[0m 0.7557  \u001b[0m | \u001b[0m 96.6    \u001b[0m | \u001b[0m 0.7212  \u001b[0m | \u001b[0m 10.31   \u001b[0m | \u001b[0m 9.808   \u001b[0m | \u001b[0m 36.71   \u001b[0m | \u001b[0m 0.184   \u001b[0m | \u001b[0m 10.74   \u001b[0m | \u001b[0m 6.339   \u001b[0m | \u001b[0m 9.232   \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=51, subsample_freq=0 will be ignored. Current value: bagging_freq=51\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=51, subsample_freq=0 will be ignored. Current value: bagging_freq=51\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=51, subsample_freq=0 will be ignored. Current value: bagging_freq=51\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=51, subsample_freq=0 will be ignored. Current value: bagging_freq=51\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=51, subsample_freq=0 will be ignored. Current value: bagging_freq=51\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "| \u001b[0m 13      \u001b[0m | \u001b[0m 0.7298  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 51.7    \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 3.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 55.9    \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 10.0    \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=12, subsample_freq=0 will be ignored. Current value: bagging_freq=12\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=62, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=62\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "| \u001b[0m 14      \u001b[0m | \u001b[0m 0.7312  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 12.86   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 20.0    \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 62.21   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 55.12   \u001b[0m | \u001b[0m 0.9886  \u001b[0m | \u001b[0m 10.0    \u001b[0m |\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=64, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=64\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=64, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=64\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=64, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=64\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=64, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=64\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "[LightGBM] [Warning] feature_fraction is set=1.0, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=1.0\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=64, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=64\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.5, subsample=1.0 will be ignored. Current value: bagging_fraction=0.5\n",
      "| \u001b[0m 15      \u001b[0m | \u001b[0m 0.7296  \u001b[0m | \u001b[0m 0.5     \u001b[0m | \u001b[0m 37.58   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 3.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 64.9    \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 47.53   \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 0.0     \u001b[0m |\n",
      "=================================================================================================================================================\n"
     ]
    }
   ],
   "source": [
    "# we need roll back to scipy==1.7\n",
    "# https://github.com/fmfn/BayesianOptimization/issues/300\n",
    "bayes_lgb.maximize(n_iter=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7a962c85-c2ea-4621-a267-33adc91682ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'target': 0.7318286646522262,\n",
       " 'params': {'bagging_fraction': 0.841721472291185,\n",
       "  'bagging_freq': 54.592298927681746,\n",
       "  'feature_fraction': 0.6385920673933405,\n",
       "  'max_depth': 12.04978395140528,\n",
       "  'min_child_weight': 9.392496953196893,\n",
       "  'min_data_in_leaf': 51.3017845966266,\n",
       "  'min_split_gain': 0.47736324501182315,\n",
       "  'num_leaves': 34.643935032031465,\n",
       "  'reg_alpha': 6.3523692278351005,\n",
       "  'reg_lambda': 8.733053317385387}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes_lgb.max"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a45b3c1a-715e-4033-b357-c2152e20a74c",
   "metadata": {},
   "source": [
    "### K-Fold Cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c4de8f11-a59a-4a41-974c-8b8ec0a86e44",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cv_model(clf, train_x, train_y, clf_name, best_params):\n",
    "    folds = 5\n",
    "    seed = 2020\n",
    "    kf = KFold(n_splits=folds, shuffle=True, random_state=seed)\n",
    "    train = np.zeros(train_x.shape[0])\n",
    "    cv_scores = []\n",
    "    for i, (train_index, valid_index) in enumerate(kf.split(train_x, train_y)):\n",
    "        print('************************************ Round {} ************************************'.format(str(i+1)))\n",
    "        trn_x, trn_y, val_x, val_y = train_x.iloc[train_index], train_y[train_index], train_x.iloc[valid_index], train_y[valid_index]\n",
    "        \n",
    "        if clf_name == \"lgb\":\n",
    "            train_matrix = clf.Dataset(trn_x, label=trn_y)\n",
    "            valid_matrix = clf.Dataset(val_x, label=val_y)\n",
    "\n",
    "            params = {\n",
    "                'boosting_type': 'gbdt',\n",
    "                'objective': 'binary',\n",
    "                'metric': 'auc',\n",
    "                'min_child_weight': best_params['min_child_weight'],\n",
    "                'num_leaves': int(best_params['num_leaves']),\n",
    "                'lambda_l1': best_params['reg_alpha'],\n",
    "                'lambda_l2': best_params['reg_lambda'],\n",
    "                'feature_fraction': best_params['feature_fraction'],\n",
    "                'bagging_fraction': best_params['bagging_fraction'],\n",
    "                'bagging_freq': int(best_params['bagging_freq']),\n",
    "                'learning_rate': 0.1,\n",
    "                'seed': 2022,\n",
    "                'nthread': 28,\n",
    "                'n_jobs':24,\n",
    "                'silent': True,\n",
    "                'verbose': -1,\n",
    "                \n",
    "            }\n",
    "\n",
    "            model = clf.train(params, train_matrix, 50000, valid_sets=[train_matrix, valid_matrix], verbose_eval=200, early_stopping_rounds=200)\n",
    "            print(\"Feature Importance:\\n\", list(sorted(zip(trn_x.columns, model.feature_importance(\"gain\")), key=lambda x: x[1], reverse=True))[:30])\n",
    "            val_pred = model.predict(val_x, num_iteration=model.best_iteration)            \n",
    "        else:\n",
    "            raise NotImplementedError('Unsupported classifer {}'.format(clf_name)) \n",
    "            \n",
    "        train[valid_index] = val_pred\n",
    "        cv_scores.append(roc_auc_score(val_y, val_pred))\n",
    "    \n",
    "    print(\"%s_score_train_list:\" % clf_name, cv_scores)\n",
    "    print(\"%s_score_mean:\" % clf_name, np.mean(cv_scores))\n",
    "    print(\"%s_score_std:\" % clf_name, np.std(cv_scores))\n",
    "    return train\n",
    "\n",
    "def lgb_model(X_train, y_train, best_params):\n",
    "    lgb_train = cv_model(lgb, X_train, y_train, \"lgb\", best_params)\n",
    "    return lgb_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9277ca86-e5ca-4799-a8f6-0520a081c5f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "************************************ Round 1 ************************************\n",
      "[LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.748213\tvalid_1's auc: 0.730123\n",
      "[400]\ttraining's auc: 0.762271\tvalid_1's auc: 0.730685\n",
      "Early stopping, best iteration is:\n",
      "[330]\ttraining's auc: 0.757732\tvalid_1's auc: 0.730799\n",
      "Feature Importance:\n",
      " [('subGrade', 71254.78851079941), ('issueDateDT', 21611.031715273857), ('grade_to_std_n7', 18241.14588224888), ('grade_to_mean_n4', 16897.05391216278), ('term', 14653.579342603683), ('grade_to_mean_n7', 13450.159541726112), ('grade_to_std_n4', 11264.357944369316), ('annualIncome', 9850.464185237885), ('dti', 9529.163885712624), ('homeOwnership', 9188.58430826664), ('subGrade_target_mean', 8472.05193889141), ('loanAmnt', 7373.37351167202), ('revolBal', 6915.743562698364), ('employmentTitle', 6291.324475646019), ('regionCode', 5682.361752152443), ('installment', 5089.7564042806625), ('grade_to_std_n8', 4489.833872437477), ('ficoRangeLow', 4265.921775102615), ('revolUtil', 3941.998643040657), ('earliesCreditLine', 3679.712715148926), ('interestRate', 3574.6939536333084), ('n2', 3444.9916689395905), ('n14', 3386.935166835785), ('totalAcc', 3099.3166880607605), ('postCode', 2740.6233369112015), ('ficoRangeHigh', 2734.0261323451996), ('n6', 2282.9953696727753), ('grade_to_std_n1', 1981.2872714996338), ('grade_to_mean_n5', 1855.1546008586884), ('grade_to_mean_n10', 1838.2569375038147)]\n",
      "************************************ Round 2 ************************************\n",
      "[LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.747822\tvalid_1's auc: 0.731279\n",
      "[400]\ttraining's auc: 0.761783\tvalid_1's auc: 0.731995\n",
      "Early stopping, best iteration is:\n",
      "[393]\ttraining's auc: 0.761339\tvalid_1's auc: 0.732036\n",
      "Feature Importance:\n",
      " [('subGrade', 81517.501704216), ('issueDateDT', 22239.46546316147), ('grade_to_mean_n10', 20276.02507865429), ('grade_to_mean_n4', 18239.5979231596), ('term', 14463.682860732079), ('annualIncome', 10282.303616166115), ('dti', 10135.59547829628), ('homeOwnership', 9106.720373392105), ('grade', 8758.966969966888), ('grade_to_mean_n7', 8704.291876792908), ('employmentTitle', 7418.254672408104), ('subGrade_target_mean', 7356.1354167461395), ('loanAmnt', 7205.694467306137), ('revolBal', 6963.921489357948), ('regionCode', 6114.97308909893), ('installment', 5337.582589149475), ('ficoRangeLow', 4566.919164419174), ('revolUtil', 4379.029355406761), ('earliesCreditLine', 3925.1267228126526), ('postCode', 3459.412938475609), ('n2', 3403.795484185219), ('n14', 3277.9441154003143), ('totalAcc', 3211.86307489872), ('interestRate', 3197.5672796964645), ('grade_to_std_n8', 2709.6421439647675), ('ficoRangeHigh', 2481.7377893924713), ('employmentLength', 2361.2740679979324), ('n6', 2256.02882540226), ('grade_to_std_n7', 2080.308973312378), ('title', 1834.183591723442)]\n",
      "************************************ Round 3 ************************************\n",
      "[LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.747655\tvalid_1's auc: 0.732825\n",
      "[400]\ttraining's auc: 0.761408\tvalid_1's auc: 0.734345\n",
      "[600]\ttraining's auc: 0.77374\tvalid_1's auc: 0.734364\n",
      "[200]\ttraining's auc: 0.747598\tvalid_1's auc: 0.733263\n",
      "[400]\ttraining's auc: 0.761936\tvalid_1's auc: 0.733871\n",
      "[600]\ttraining's auc: 0.774333\tvalid_1's auc: 0.734105\n",
      "Early stopping, best iteration is:\n",
      "[466]\ttraining's auc: 0.766051\tvalid_1's auc: 0.734233\n",
      "Feature Importance:\n",
      " [('subGrade', 74897.2998880148), ('issueDateDT', 22634.668679594994), ('grade_to_mean_n4', 22156.124026060104), ('grade_to_mean_n7', 17530.977856516838), ('grade_to_std_n7', 16648.771657824516), ('term', 14120.97695851326), ('dti', 10806.359160542488), ('annualIncome', 10772.092508554459), ('homeOwnership', 9541.53991651535), ('loanAmnt', 8286.500488519669), ('revolBal', 7385.197965860367), ('employmentTitle', 7245.033922433853), ('regionCode', 6836.008635759354), ('subGrade_target_mean', 5829.487451553345), ('installment', 5605.744664907455), ('ficoRangeLow', 5274.916856169701), ('revolUtil', 4843.551537036896), ('earliesCreditLine', 4266.445522546768), ('grade_to_std_n1', 4171.80390048027), ('totalAcc', 4130.674059152603), ('postCode', 4012.612011194229), ('n14', 3677.4475779533386), ('interestRate', 3488.8201075792313), ('n2', 3315.736514568329), ('grade_to_mean_n5', 3081.134559750557), ('grade_to_mean_n10', 2772.2625319957733), ('grade_to_std_n8', 2576.7370405197144), ('n6', 2480.334767103195), ('employmentLength', 2475.512908577919), ('ficoRangeHigh', 2472.23561835289)]\n",
      "lgb_score_train_list: [0.7307985054855854, 0.7320358167416893, 0.734461353385744, 0.7294853358957647, 0.7342331687404857]\n",
      "lgb_score_mean: 0.7322028360498538\n",
      "lgb_score_std: 0.0019291447981775595\n"
     ]
    }
   ],
   "source": [
    "best_params = bayes_lgb.max['params']\n",
    "lgb_train = lgb_model(X_train, y_train, best_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2e669f1-c310-45d4-be9a-2f5e084a04c1",
   "metadata": {},
   "source": [
    "### Retrain the model on the full training dataset\n",
    "In this section, we should use the full training dataset and evaluate the model effect on the test dataset. However, since we do not have labeled test dateset, we can only use the same training settings as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a352e627-0008-49ba-9cec-3e1594417fe1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.748111\tvalid_1's auc: 0.729737\n",
      "[400]\ttraining's auc: 0.762032\tvalid_1's auc: 0.73047\n",
      "Early stopping, best iteration is:\n",
      "[325]\ttraining's auc: 0.757318\tvalid_1's auc: 0.730558\n",
      "Feature Importance:\n",
      " [('subGrade', 70057.59370136261), ('issueDateDT', 21630.305247068405), ('grade_to_mean_n4', 20378.347585082054), ('grade_to_std_n7', 20268.76363682747), ('grade_to_mean_n7', 16803.37327694893), ('term', 14263.790437221527), ('grade_to_std_n4', 11528.269649267197), ('annualIncome', 9617.149665117264), ('homeOwnership', 9208.273541092873), ('dti', 8944.923849582672), ('loanAmnt', 6901.981902122498), ('employmentTitle', 6674.215544462204), ('revolBal', 6415.410744309425), ('regionCode', 5876.525661706924), ('subGrade_target_mean', 5788.768405795097), ('installment', 5115.87585067749), ('ficoRangeLow', 4422.903689146042), ('grade_to_mean_n10', 4343.6329555511475), ('n2', 3731.807690858841), ('revolUtil', 3568.5567647218704), ('totalAcc', 3430.6784932613373), ('n14', 3286.1945281028748), ('earliesCreditLine', 3230.568236231804), ('interestRate', 2912.0687049627304), ('postCode', 2806.5340764522552), ('grade_to_std_n8', 2507.1707491874695), ('ficoRangeHigh', 2082.7660686969757), ('n6', 2046.4387538433075), ('employmentLength', 1766.163073182106), ('n8', 1630.0977778434753)]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "train, valid = train_test_split(train_data, test_size=0.2, random_state=1)\n",
    "X_train, y_train = train[feature_cols], train[label]\n",
    "X_valid, y_valid = valid[feature_cols], valid[label]\n",
    "train_matrix = lgb.Dataset(X_train, label=y_train)\n",
    "valid_matrix = lgb.Dataset(X_valid, label=y_valid)\n",
    "\n",
    "params = {\n",
    "    'boosting_type': 'gbdt',\n",
    "    'objective': 'binary',\n",
    "    'metric': 'auc',\n",
    "    'min_child_weight': best_params['min_child_weight'],\n",
    "    'num_leaves': int(best_params['num_leaves']),\n",
    "    'lambda_l1': best_params['reg_alpha'],\n",
    "    'lambda_l2': best_params['reg_lambda'],\n",
    "    'feature_fraction': best_params['feature_fraction'],\n",
    "    'bagging_fraction': best_params['bagging_fraction'],\n",
    "    'bagging_freq': int(best_params['bagging_freq']),\n",
    "    'learning_rate': 0.1,\n",
    "    'seed': 2022,\n",
    "    'nthread': 28,\n",
    "    'n_jobs':24,\n",
    "    'silent': True,\n",
    "    'verbose': -1,\n",
    "\n",
    "}\n",
    "\n",
    "model = lgb.train(params, train_matrix, 50000, valid_sets=[train_matrix, valid_matrix], verbose_eval=200, early_stopping_rounds=200)\n",
    "print(\"Feature Importance:\\n\", list(sorted(zip(X_train.columns, model.feature_importance(\"gain\")), key=lambda x: x[1], reverse=True))[:30])  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eec1bee1-da64-4fb3-a5cb-7092c4e38025",
   "metadata": {},
   "source": [
    "### Acknowledgement\n",
    "Thanks to the Tianchi community for providing the loan default dataset and corresponding tutorial for risk management based on this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3d2852f-1438-401d-b430-ac76a206fdcf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
